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ABSTRACT 

This study examines the effects of reflection category and reflection quality on learning outcomes during 
Web-based portfolio assessment process. Experimental subjects consist of forty-five eight-grade students in a 
“Computer Application” course. Through the Web-based portfolio assessment system, these students write 
reflection, and join self-assessment and peer-assessment. The Phrase Processing System is used to distinguish 
phrases in students’ reflections, and finally students’ reflections are classified. Results from this research indicate 
that the effect size of reflection category on learning outcomes measured by achievement test, work, and attitude 
is extremely small and not significant. The effect size of reflection quality on learning outcomes was small, but 
significantly positive. Follow-up contrasts found reflection quality significantly related to achievement test, work, 
and attitude outcomes. 

Keywords: Portfolio, Portfolio Assessment, Reflection, Learning outcome 

INTRODUCTION 

Portfolio Assessment is a formative process based on the content of an individual student’s portfolio. It aims to 
review the achievement and capability which students have obtained from their learning. Moreover, it assists 
students in solving the difficulties they encounter during the learning process. What students are creating a 
portfolio for is to have a sense of accomplishment, which may induce learners to have a feeling of honor, 
responsibility and contribution (Paris and Ayres, 1994). An e-portfolio served as a record keeping tool has been 
widely used to help preservice teachers develop their reflective skills (Hemer-Patnode & Lee, 2009; Yao, 
Aldrich, Foster, & Pecina, 2009). Grant, Vermunt, Kinnersley, and Houston (2007) used portfolio assessment in 
reflection activities of medical college freshmen. Eppink (2002) proposed that portfolio assessment might 
enhance a student’s self-reflection and promote the development of meta-cognition. Rees, Shepherd, and 
Chamberlain (2005) used a reflective portfolio for assessing a medical college student’s professional 
development and encouraging his/her reflective thinking. Therefore, portfolio assessment helps develop a 
student's reflective thinking skill and diversified intellectual activities, and provides students with more effective 
and practical evaluation. 

In recent years, portfolios and reflection journals concerning the development of a learner’s reflection capability 
have received increasing attention. Chang (2008) argued in his study that the implication of portfolio assessment 
should involve a learner’s reflection, with an intention to allow the learner to review his/her own learning 
process. He further argued that portfolio assessment also allowed the learner to identify a learning method that 
best fit his/her learning by reflecting his/her failure experience and thereby promote his/her lifelong learning 
capability. Lazear (1999) suggested that reflection was the core for a learner in the learning process. For this 
reason, reflection is an indispensable element of a portfolio (Barrett & Garrett, 2009; Barrett, 2010). The 
difference between a portfolio and a folder lies in the learner’s reflection; without a learner’s reflection, a 
portfolio is nothing but another type of folder. According to Barrett (2004), an e-portfolio without reflection is 
just a media document file, a magic electronic resume, or a digitalized clipboard. Therefore, we can create a 
reflection-focused portfolio by adding appropriate reflection and feedback, such as learning journal, 
self-assessment, peer-assessment and feedback, into Web-based portfolio assessment. 

Reflection can be categorized into reviewing process, contemplative process, comparing process and judging 
process (Santos, 1997). By degree, the quality of reflection can be evaluated and classified into “unmatchable,” 
“basic,” “good” and “outstanding” (Morgan, 1999). A study of Chen, Kinshuk, Wei, and Liu (2010) evaluated 
reflection quality based on the five levels which are reporting, responding, relating, reasoning, and 
reconstructing. Additionally, it can also be divided into “purpose,” “supportability,” “systematization,” “syntax” 
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and “writing skill” (King-Shaver, 1999). Learners will show different reflection qualities in accordance with 
different levels and types. However, what are the levels or categories of reflection occurring to learners during 
the process of Web-based portfolio assessment? How will reflection be categorized? How is the reflection 
quality during Web-Based portfolio assessment process? How can it be measured? These are issues worthy of 
study and concern. 

In terms of learning outcome promoted by reflection, it, for instance, can boost a medical college junior’s 
performance on the skill test of clinical diagnosis (Blatt, Plack, Maring, Mintz, & Slmmens, 2007). Besides, 
reflection may also increase a business school freshman’s self-regulated learning ability and academic 
performance (Masui & De Corte, 2005). Writing a reflective journal may develop higher scientific literacy of 
pre-service teachers (Gibson, Bernhard, Kropf, Ramirez, & Van Strat, 2001). Reflection may facilitate a student’s 
review and correction of his initial ideas and therefore leads to more acceptable work (Davis, 2000). In 
promotion of attitude and meta-cognition, students who gradually understand their learning role during the 
course of reflection can have an insight into their thinking process and their characteristics, attitude, attention, 
dominance, persistence, and other fundamental responsibilities (L. Campbell, B. Campbell, and Dickinson, 
2002). King-Shaver (1999) suggested that the most significant benefit of reflection was to allow learners to 
understand their merits and limitations on learning and ultimately take responsible for their learning behavior. 

Most of the studies on the effect of reflection as discussed above involve the comparison between learners with 
reflection and without reflection and have nothing to do with the effect of reflection category and quality on 
learning outcome. Most reflections are observed in the classroom, without involving the use of a portfolio, not to 
mention the use of Web-based portfolio assessment. If we classify reflection into some levels and then use 
statistics to compare the difference in learning outcome between learners with different categories of reflection, 
we may judge whether the categories of reflection have any effect on learning outcome. Alternatively, if we 
assess and classify reflection and then use statistics to compare the difference in learning outcome between 
learners with different reflection qualities, we may judge whether different reflection qualities have any effect on 
learning outcome. However, during the course of Web-based portfolio assessment, is there any difference in 
learning outcome between learners with different reflection qualities? Does a learner with better reflection 
quality also have better learning outcome? Is there any difference in learning outcome between learners with 
different categories of reflection during Web-based portfolio assessment process? These issues are the ones the 
study intends to explore. 

Based on the foregoing context, this study is designed to examine the effect of learners’ reflection behavior 
(category and quality) on learning outcome during Web-based portfolio assessment process. The research 
questions are as follows: 

1. Are there significant differences in the learning outcome (measure by the scores on achievement tests, the 
rubric score of work submissions, and the score on an attitude survey), based on the categories (emotion, 
memory, cognition, and evaluation lexicons) of self-reflections in a web-based portfolio assessment? 

2. Are there significant differences in the learning outcome, based on the qualities (high, middle, and low) of 
self-reflections in a web-based portfolio assessment? 

LITERATURE REVIEW 

Web-Based portfolio assessment and Online Reflection 

A Web-based portfolio assessment system developed by Chang (2008) includes goal setting, reflection writing, 
work uploading and demonstration, self-assessment, peer-assessment, and teacher assessment and feedback, etc. 
Wielenga, Ritzen, and Kosters (1999) developed a Web-based portfolio assessment system for an experiment 
with pre-service teachers. The system has functions like personal profile: where the learner’s experience and 
background can be edited, modified and browsed; Work storage: the learner’s work can be saved; 
Self-assessment: the learner can assess or defend his/her learning achievement here; Work demonstration: the 
learner’s work and reflection can be demonstrated; Portfolio browsing: the learner’s portfolio can be browsed; 
Online records: the learner’s learning situation and process in the system can be recorded. 

From the forgoing discussion, it is found that the Web-based portfolio assessment system must include learners’ 
reflection on learning, peer discussion, peer-assessment and feedback, self-assessment, and dialogs between 
teachers and students to encourage reflection of students (Eppink, 2002), so as to match the implication of 
Web-based portfolio assessment. The interaction between teachers and students or between peers and feedback 
may serve as reference for self-reflection and correction for achieving the learning goals. Online reflection is 
crucial for Web-based portfolio assessment; it can (l)increase reflection ability (Avrramidou & Zembal-Saul, 
2002; Coombe & Barlow, 2004; Morris & Buckland, 2000); (2)increase a learner’s critical thinking ability; 
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(3)facilitate writing, storage, modification, browsing and inspection of reflection; (4)facilitate an instructor’s and 
learner’s inquiry, review and comparison (Chang & Tseng, 2009a; 2009b). From the argument given above, 
online reflection is not only convenient for learners to proceed with reflection but also helpful for learning. 

Effects of reflection 

Regarding the effects of reflection on learning achievement, Costa and Kallick (2000) argued that a learner who 
was more likely to proceed with reflection would better control his/her thinking and inference and will therefore 
have better communications with peers and teachers. The inference performance has also been confirmed in 
other studies. Murphy (2004) selected nursing school freshmen as study subjects and found that the inference 
performance of students with reflection activities was superior to that of students without reflection activities. 
Yancey (2001) suggested that the reason why advocates supported the selection of portfolio was they believe a 
learner could review how he/she overcame learning difficulties when writing reflection and therefore could 
improve an individual’s learning outcome. In conclusion, reflection is helpful for thinking, inference, diagnosis 
and learning. 

Regarding the effects of reflection on attitude and meta-cognition, Saito and Miwa (2007) found in their study 
that the experimental group with reflection activities is superior to the control group in data collection. Gama 
(2004) developed a solution for environmental issues with algebra and incorporates e-reflection assistant. 
According to his study, reflection provided by the e-reflection assistant can improve learning outcome, time 
management skill, and knowledge application ability. Irby and Brown (1999) supported that reflection, if carried 
out continuously, could boost the performance of self-managing tasks. These studies show that reflection may 
enhance data collection performance, time management skill, knowledge application ability, learning ability, and 
learning attitude, etc. 

From the discussion provided above, reflection has a wide range of effects which can be observed from some 
aspects. This study assesses reflection effects from three aspects: achievement test, work and attitude. 
Achievement test can indicates a student’s learning achievement; work is a necessary item in a learning portfolio; 
and attitude shows a student’s meta-cognitive abilities. 

Assessment of Reflection 

Categories of reflection 

Chirema (2007) analyzed reflective journals to determine the three categories of reflection: non-reflection, 
reflection, and critical reflection. Wood, King, Kitchener, and Lynch (1994) and Wood (2000) also proposed 
three categories of reflection: pre-reflective thinking, quasi-reflective thinking, and reflective thinking, which 
may be used to differentiate between learners with different levels of reflection. These studies unveil that the 
reflection levels of senior high school students concentrate on pre-reflection. Santos (1997) interviewed 28 
learners and studies their cognition on reflection. Fie found that a learner can reflect on their learning from 
reviewing process - the learner prevents his/her mistake from recurring by reviewing past works; contemplative 
process - the learner spends time contemplating himself/herself; comparing process - the learner compares the 
goal established initially and the ultimate result to determine whether the expected goal is achieved; and judging 
process - the learner evaluates the merits and demerits during self-learning. In short, reflection can be 
categorized into inspection, thinking, contrast, and evaluation. 

By the nature of reflection content, Wang (2002) classified reflection into descriptive reflection, dialogic 
reflection, and critical reflection. Based on undertaking of reflection, Grossman (2009) categorized reflection 
into content-based reflection, meta-cognitive reflection, self-authorship reflection, and transformative or 
intensive reflection. Lee (2002) suggested that reflection was a psychological activity of meta-cognition and 
could be divided into three types: cognition, comparison, and evaluation. By characteristics, Lin (2004) divided a 
learner’s reflection into emotional reaction, cognition and combination. In her study, Lin also found that learners 
would express their feeling and emotion by selecting terms which they frequently use. Therefore, it is possible to 
further explore a learner’s reflection by characteristics of his/her behavior (i.e. status of cognition, evaluation, 
memory, or emotion). 

With different study purposes, study processes, inductive methods and learner level, the designations of 
categories of reflection are not the same; however, there are several common designations such as emotion, 
memory, cognition, contrast, evaluation, and combination, which can serve as the basis for categorization in this 
study. The contrast category of reflection indicates the comparison, contrast and review of learning between the 
learner himself/herself versus his/her peers. The evaluation category of reflection indicates comments, 
measurement and criticism in addition to comparison, contrast and review. As evaluation includes the 
implication of contrast, we may incorporate the category of contrast into the category of evaluation in case of 
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few study subjects. 

Assessment Rubrics of Reflection 

Teachers at South Brunswick Schools, New Jersey, USA used rubrics of reflection evaluation for assessing 
reflection writing in student portfolios. The assessment includes the following items: reflection objectives - clear 
descriptions of key points and suggestions and listing the importance of well defined themes or tasks; 
supportability - clear descriptions of evidence provided and effective demonstration of major arguments; 
organization - clear introduction, together with logic and clear inference for conceptual development; syntax - 
well-organized syntax, namely sentences in proper length without error; wording and diction - strong wording 
with a wide range of diction; and writing skill - correct spelling, correct use of punctuation, capital letters, and 
grammar, without obvious errors (King-Shaver, 1999). The first two items are highlighted as key points and 
evidence; the other four are about writing (even the first two are also associated with writing). The emphasis on 
writing skills will give high scores of reflection to those who have better writing abilities or skills. In addition, 
learning progress, learning outcomes, learning attitude and peer feedback are not substantially included, which is 
an inadequacy in that assessment model. 

According to Tomkinson (2002), items used to examine reflection may include learning achievement, progress 
with learning tasks, defects of progress with whole programs, current learning situation, merits or demerits from 
other’s feedback, chances for re-choice, chances missed, reasons to success, reasons to failure, learning 
difficulties, required development, required support, support sources, learning sources, and future plans. These 
items seem chaotic; it would be better if they can be classified into some aspects or constructs. Sparks-Langer, 
Simmons, Pasch, Colton, and Starko (1990) argued that questions or solutions proposed by learners were 
indispensable elements, because learners would not know what to do next if they do not understand what 
problems they encounter or how to solve the problems. 

From the foregoing argument, the rubrics of reflection evaluation have two major aspects: writing skill and 
content quality. Content quality includes key points, evidence and examples, learning achievement, learning 
situation or experience, feedback from others, chances for learning or growing, reasons to success or failure, 
learning difficulties and solutions, required development and support, future plans, and improvement, etc. There 
are diversified evaluation indicators for content quality. As the assessment items listed above do not address 
reflection in portfolios, they do not involve the content items (or entries) of a portfolio, e.g. reflection on learning 
goals, reflection after reviewing peers’ portfolios. This study examines reflection written by students when they 
are creating their portfolios, rather the reflection without portfolio. Thus, the evaluation of reflection quality in 
this study will integrate the above-mentioned evaluation rubrics (i.e. reflection on learning outcomes, reflection 
on learning attitude, reflection on the feedback from peers, improvement) and reflection based on portfolio 
entries. 

METHOD 

Subjects 

This study targets at 45 eight-grade students (24 males and 21 females) as subjects who study a course of 
“Computer Application” at some junior high school in the Taiwan. The average of the student age is 14. The 
duration of the study was a 10-week period with 3 hours for each week. The students may spend more time 
creating their works except for classroom time. The students have not been involved in a Web-based portfolio 
assessment system before the study. Through the system, these students write reflection, and join self-assessment 
and peer-assessment. Their teacher conducts reflection teaching and various activities such as review and 
assessment of student portfolios. These students have sufficient computer operations and Internet skills for using 
the system since they have learned computer for one year. 

Learning contents are based on two course units “Computer Animations” and “Time Axis Control” in the 
textbook, and works are created by Photoimpact and Dreamweaver MX. As students have to submit their 
digitalized works in the two course units, the system is ideal for review of the works and learning processes. The 
students were not informed that they were participating in a study in order to avoid the so-called Hawthorne 
effect and John Henry effect. 

Research Framework 

This study conducts phrase processing for the reflection content with the Phrase Processing System developed 
by the Institute of Information Science (2007), Academia Sinica and then proceeds with organization and 
categorization based on the results of phrase processing, and ultimately uses MANOVA to test whether there is 
any significant difference in learning outcome (achievement test, work and attitude) between learners with 
different categories of reflection and different reflection qualities, and further verifies the statistical results by the 
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reflection in learners’ portfolios. Research framework is shown as Figure 1, while research variable are as 
follows. 

1. Categories of reflection, which are obtained from phrase processing results and analysis. They include 
emotion, memory, cognition, evaluation, and combination. The emotional category of reflection represents 
the learner’s description of his/her learning or his/her peers’ learning; it is simple emotional reaction without 
in-depth descriptions. Mnemonic reflection means only the learner’s description of his/her learning or his/her 
peers’ learning, without further review. The cognitive category of reflection indicates the learner’s review of 
his/her learning or his/her peers’ learning, while no further comment or criticism is made. The evaluation 
category of reflection indicates the learner’s comment or criticism on his/her learning or his/her peers’ 
learning. The combination category of reflection means a minimum inclusion of any two categories 
described above. These categories of reflection do not involve the merits and demerits of reflection and are 
classified as nominal variables in statistical application. 

2. Reflection qualities are rated based on the content of reflection measured by a reflection questionnaire (see 

details in the research instrument section). 

3. Learning outcomes include scores of achievement test, work and attitude. The scores of a learner’s work and 

attitude are average scores given for those two course units and are measured by the questionnaire of 
portfolio assessment in the Web-based portfolio assessment system. The score of achievement test refers to 
the score that a learner earns in the paper test at the end of the course. 



Figure 1 Research framework 


Procedure and Activities 

Preparation State (1st week) 

The teacher teaches the concept, assessment methods and reflection writing skills of a portfolio to the students in 
the classroom. Besides, the teacher also demonstrates the use of the Web-based portfolio assessment system and 
the questionnaire of portfolio assessment. At the last part, learners may try using the system so that they may 
have an understanding of the system’s functions and operations. 

State of Course Unit #1 (2nd week to 5th week) 

The teacher teaches Unit #1 (Computer animations) in line with the use of the Web-based portfolio assessment 
system. Learners may take advantage of spare time to participate various activities, such as individual portfolio 
creation (setting learning goals, online uploading works, and writing reflection, etc.) by form filling, viewing 
peers’ portfolios, self-assessment, peer-assessment, and online discussion etc., in the Web-based portfolio 
assessment system. This process is known as portfolio assessment. Besides, the teacher and the online assistant 
use the questionnaire of portfolio assessment developed by Wu (2008) to evaluate the students’ learning outcome 
(including work, reflection and attitude) based on their portfolio contents and online behavior performance at the 
end of the unit. 

The student, together with a number of peers assigned by the teacher, uses the questionnaire of portfolio 
assessment for anonymous peer-assessment. Moreover, the teacher will offer guidance and answer questions 
raised by students regarding the use of the portfolio assessment and system on the discussion board within the 
system and during class. As for the activities of creating the works that are in students’ portfolios, the assessment 
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processes of the works are formative. Students have an opportunity to improve their works after receiving 
feedbacks. 

Stage of Course Unit #2 (6th week to 9th week) 

At this stage the teacher teaches Unit #2 (Time axis control) and repeats the activities described in Unit #2. Prior 
to the beginning of the stage, the teacher gives further guidance on the problems which learners have faced in the 
previous stage such as setting learning goals, writing reflection, uploading works, self-assessment, 
peer-assessment, use of portfolio assessment questionnaire. 

Stage of Achievement Test (10th week) 

There is an achievement test at the end of the two course units. 

Procedure of Writing Reflection 

During the course of the experiment, learners have to create their portfolios (including setting learning goals, 
uploading works, and writing reflection), and proceed with portfolio viewing and sharing, self-assessment, and 
peer-assessment, with the assistance of the Web-based portfolio assessment system. The procedures for writing 
reflection are provided below and the screen shots are shown as Figure 2. 

1. The teacher explains the outline of reflection in the system to learners to help them write reflection. 

2. Regarding the sequence of creating a portfolio, a learner first sets learning goals. After finishing the work, the 

learner writes reflection in accordance with outline of reflection provided by the system. The outline of 
reflection includes reflection on learning goals, reflection on learning outcomes, reflection on learning 
attitude, reflection on peer performance, and reflection on feedbacks. 

3. When a learner writes “reflection on learning goals”, he/she may review his/her learning goal established 

initially in the “Portfolio Creation” area, based on which the learner can write reflection. 

4. When a learner intends to write reflection for “reflection on peer performance”, he/she may browse their 

portfolios in the “Portfolio Assessment” area, view their online participation record and browse 
self-assessment, peer assessment and teacher assessment in the “Portfolio Scores” area; after comparison 
with peers’ performances, the learner can write reflection. 

5. When writing reflection for “reflection on feedbacks”, a learner may think peers’ comments or browse peers’ 

feedbacks in the “Portfolio Assessment” area in advance. 

6. When finishing writing reflection, the learner may click the “Send Data” button, and the system will 
automatically save the reflection in a personal portfolio. The learner may go to the “Portfolio Assessment” 
area to browse his written reflection or peers’ reflection. In addition, the leaner may repeat the previous 
actions to write reflection for different thinking or outlines, if he/she feels that the content of reflection 
should be supplemented or re-written. 



Site URL : Portfolio Index > Portfolio Creation> S elf-reflection Writing 


I Guidance I Portfolio Creation I Portfolio Assessment I Portfolio Scores I Course Overview I L 

Self-ie0ectiom Writing 


Learning Goals 


Learning Goal 


Reflection 


Other Entries 


1. The student properly establishes pragmatic goals and anticipated outcomes based? on his/her 
ability, self-expectations, etc: 

Ans: I‘ d like to acquire useful knowledge and skills, and hopefully I' 11 make use of these understandings 
in future studies. 

2. The student ac h ieves teaching goals related to the academic subject: 

Ans: I' 11 make progress by closely following the teacher's instructions. 

3 . The student shows progress and improvement in learning attitudes and academic pe rf or m a n ce: 

Ans: I‘ 11 actively participate in class discussions so that I can learn more from other students. 

4. The student achieves personal learning goals: 

Ans: I' 11 submit the class assignments on time. 

5 . The student’ s overall performance in terms of learning goals: 

Ans: I' 11 try my best to improve every as p ect of learning. 


1. Write your reflections on personal learning goals: 

2. Illustrate your weaknesses, learning achievements 
and progress: 

3 . Write your reflections on learning attitudes and 
academic growth: 

4. Establish your self-expectations after observing 
peer performance: 

5. Offer explanations and ways for improvement based 
on teacher or peer feedback: 


| submit | [ Clear 


Figure 2 The screen shots of writing reflection 
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Research Instrument 

Web-Based Portfolio assessment System 

This study uses a self-developed Web-based portfolio assessment system for conduct of an experiment. 
Functionality of the system includes: 1. Guidelines for portfolio creation; 2. Portfolio creation: filling basic 
information, setting learning goal, online uploading works, writing reflection, and other content creation (e.g. 
anecdote, Website sharing, e-document sharing, achievement testing outcomes, or other entries, etc.); 3. Portfolio 
assessment: (1) may be distinguished by the student’s name, work title, or work sample; (2) can be divided into 
teacher assessment, self-assessment, and peer-assessment, by the log-in ID, all of which use the same portfolio 
assessment questionnaire; and peer portfolios can be also browsed and reviewed; 4. Portfolio scoring: including 
scores of self-assessment, scores of peer-assessment and comments, scores of teacher assessment and comments, 
overall mean (teacher may set up the rate of teacher assessment, student self-assessment, and peer-assessment); 
excellent works are highlighted; 5. Course descriptions: including the names of course units, syllabus and teacher 
profiles, etc.; 6. System management; 7.0nline discussion board (including course discussion and portfolio 
discussion); 8. Bulletin; and 9. Personal profile maintenance. 

Questionnaire of Web-Based Portfolio Assessment 

The teacher and the online assistant use the Web-based portfolio assessment questionnaire developed by Wu 
(2008) to assess students’ learning outcome based on portfolios created by the students. The questionnaire 
consists of six aspects that are portfolio creation, learning goals, works, reflection, attitude, and other, and the 
score of portfolio assessment is equal to the sum of all the scores given for the six aspects. Three aspects (work, 
reflection, and attitude) are used to represent the learning outcome (dependent variable) in this study. Other three 
aspects (portfolio creation and learning goal, and other aspects) are not used in this study. 

The scoring method of the questionnaire is given based on the performance that learners have achieved: 1, 1.5, 2, 
2.5, 3, 3.5, 4, 4.5 and 5; the higher the score, the better the performance of assessment items. 

Content of Questionnaire 

Reflection Questionnaire 

The reflection questionnaire in the portfolio assessment questionnaire is used to assess the contents of learners’ 
reflection. The assessment items include (1) reflection on learning goals: proceed with reflection for the learning 
goals set initially; (2)reflection on works: the ideas about the generation process and outcomes of works; 
reflection on learning outcomes: listing the learner’s learning achievement, demerits and progress; (3)reflection 
on learning attitude: merits/demerits of learning attitude, progress, and reflective thinking; (4)reflection on peer 
performance: self-expectations derived from the observation on peer performances; (5)reflection on feedback 
from the teacher, the online assistant and peers, and proposing improvement or explanation ; (6)overall reflection 
quality. 

Work Questionnaire 

The work questionnaire in the portfolio assessment questionnaire is used to assess the contents of learners’ works. 
The assessment items are the validity of work, appropriateness of work, integrity of work, difficulties of work, 
originality of work, the degree that learners understand learning contents, evidences of work creating process 
(e.g. ground plan of work, initial work, revised work, and etc.), and overall performance on work. 

Attitude Questionnaire 

The attitude questionnaire in the portfolio assessment questionnaire is used to assess learners’ learning attitude 
and interaction with peers. The assessment items are online viewing, browsing, peer-assessment, and feedback; 
online resource and information sharing; online discussion, knowledge sharing, idea exchange, and 
problem-solving; overall performance on attitude. 

Validity of Questionnaire 

Table 1 shows that values of KMO (Sampling proper measure of Kaiser-Meyer-Olkin) in each aspect of the 
questionnaire are larger than 0.7, reaching the standard for conduct of the factor analysis. Using its Principal 
Component Analysis (PCA) may establish validity and proceed with the orthogonal rotation using varimax 
mathod. The results of factor analysis indicated that the factor loadings of all items are greater than 0.3. 
Therefore, all items were remained. Six aspects with eigenvalues higher than 1 were extracted, but only three 
aspects were used in this study. Finally, explained variances are all larger than 70%, showing there is a high 
validity in each aspect. The overall explained variances of the two course units exceed 78%, which is similar to 
the overall explained variances (larger than 76%) of pilot test and formal test from Wu (2008). The questionnaire 
has a high validity, indicating its potential to effectively measure the quality of portfolios and learning outcomes 
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of learners. 


Table 1: Factor analysis of the Web-based portfolio assessment questionnaire 




Unit #1 


Unit #2 

Aspect 

KMO values 

Explained variances (%) 

KMO values 

Explained variances 
(%) 

Reflection 

0.89 

84.51 

0.88 

81.06 

Work 

0.80 

80.00 

0.85 

82.61 

Attitude 

0.85 

86.32 

0.86 

74.22 

Overall 

0.72 

78.91 

0.75 

86.62 


Note: The data of other aspects are not used and thus are not listed in this table. 


Reliability of Questionnaire 

All Cronbach’s a values of the questionnaire on the two units of the course are larger than 0.9, suggesting high 
internal consistency between questions in the questionnaire. These results are quite similar to the reliability of 
the questionnaire measured by Wu (2008) (The Cronbach’s a value of the pilot questionnaire test was 0.9, 
while the Cronbach’s a value of formal questionnaire test was 0.923). Table 2 demonstrates Cronbach’s a values 
of the work, reflection and attitude assessment questionnaires in the portfolio assessment questionnaire. 

Moreover, inter-rater reliability of the questionnaire shows a high consistency between the teacher and the online 
assistant by using the Pearson’s correlation (r = 0.72,/?<0.001). 


Table 2: Reliabilities of the aspects in the Web-based portfolio assessment questionnaire 


Aspect 


Cronbach’s a value 


Unit #1 


Unit #2 

Reflection 

0.819 


0.864 

Work 

0.914 


0.970 

Attitude 

0.969 


0.960 

Overall 

0.972 


0.980 


Note: The data of other aspects are not used and thus are not listed in this table. 


Achievement Test 

The achievement test, including 10 multiple-choice questions, is used to measure the degree that learners 
understand the two units of the course. The ten questions are selected from the teacher’s manual, five from Unit 
#1 and the other five from Unit #2. The Cronbach’s a value of the achievement test is 0.71, suggesting the 
consistency between the questions. Item analysis is conducted to identify the discriminatory power and 
consistency of the questions. According to the result of the t-tests, significant differences are found between the 
high and low-score groups of each question, indicating that these questions can effectively discriminate students’ 
performances. Pearson’s correlation coefficients between the questions and total scores are at the significant 
level, suggesting the consistency between the questions and the overall questionnaire. Table 3 demonstrates the 
determinant value (t value), correlation coefficient, and their corresponding significant values. 

Table 3: Item analysis of achievement test 


Item analysis 

Attribution of achievement test - 

t (Sig.) Correlation (Sig.) 


Skill of axis control 

6.01(0.00) 

0.428(0.00) 

Skill of animation 

4.34(0.00) 

0.386(0.00) 

Skill of animation 

6.35(0.00) 

0.439(0.00) 

Knowledge of axis control 

9.42(0.00) 

0.517(0.00) 

Skill of axis control 

7.12(0.00) 

0.458(0.00) 

Skill of axis control 

7.89(0.00) 

0.462(0.00) 

Knowledge of animation 

8.75(0.00) 

0.489(0.00) 

Knowledge of animation 

4.01(0.00) 

0.381(0.00) 

Skill of animation 

5.74(0.00) 

0.407(0.00) 

Knowledge of axis control 

5.02(0.00) 

0.401(0.00) 


Phrase Processing System 

This study uses the Phrase Processing System (Institute of Information Science, 2007) for phrasing the sentences 
in reflection. The lexical library in the system includes approximately 100 thousand words and phrases. With 


Copyright © The Turkish Online Journal of Educational Technology 


108 
















TOJET: The Turkish Online Journal of Educational Technology - July 2011, volume 10 Issue 3 


additional word types, term frequency, word frequency, and word-type frequency, this is the first phrase 
processing system that provides unknown word detection and syntactic category prediction. The system was 
awarded No.l of the first phrasing contest held by the International Society of Computational Linguistic. The 
system was ever used in analyzing speech communication category (Chan, 2001). With high accuracy (96%) and 
consistency (Chen & Bai, 2000), it is ideal for processing massive reflection data in this study. In addition, the 
system can increase the reliability and validity and save labor and time on content analysis of reflection. 

Besides, after phrasing sentences, the system will tag each word in student reflection. Principally, words are 
divided into dynamic verbs, situation verbs, and other word types (such as conjunction, adverb, noun, and 
pronoun, etc.). The dynamic verb means a verb which adverbial attributive can be such as very, strongly, 

extremely, .etc. A verb beyond dynamic verb means situation verb. As dynamic verbs are not suitable for 

categorization of mental state theoretically in psycholinguistics, only situation verbs are selected and serve as the 
basis for categorization of reflection (Institute of Information Science, 2007). 

The following lists the process of reflection categorization using the Phrase Process System: 

1. The contents of reflection were uploaded to the Phrase Process System. 

2. The Phrase Process System dealt with this part of work by marking up the words in the reflective journals 

with corresponding part-of-speech tags. 

3. The researchers specifically focused on various types of stative verbs, e.g. intransitive verbs, causative verbs, 
transitive verbs, etc. 

4. The researchers, with the help of Mandarin teachers and expert, worked together on grouping up vocabulary 
(i.e. stative verbs) into types, and counting the frequency usage of each type of words within a reflective 
journal. 

5. Three types of reflection were ultimately determined based on a self-developed guideline (see details in the 
Section of Results and Discussion) after the researchers coped with overlapping classification of certain words, 
and wiped out barely used vocabulary types. 

RESULT 

Categories of Reflection 

This study applies the Phrase Processing System to summarize the situation verbs used by each learner in his/her 
reflection and the occurring frequency. Afterwards, each situation verb is transferred to a mental lexicon 
(emotion, memory, cognition, or evaluation) by two linguistic experts and occurrence numbers are counted. 

The following formulation is used to verify the consistency between the two experts. In terms of the total transfer 
from situation verbs to mental lexicons for all students, the consistency between the two experts is sufficient. In 
terms of the transfer for each student from situation verbs to mental lexicons, the consistencies between the two 
experts are all sufficient. 

Percent of consistency = 2 x number of consistency / total number of transfer 
= 2 x 2139 / (2516 + 2516) = 4278 / 5032 = 0.85 

Three types of reflection were ultimately determined based on self-developed guidelines after the researchers 
coped with overlapping classification of certain words, and wiped out barely used vocabulary types. The detailed 
guidelines are as follows: (l)Cognition type refers to those reflective authors who are inclined to cognition 
words comparing to evaluation, and the rest two vocabulary types are barely occurred in this case. (2)Evaluation 
type talks about a reflective author who dominantly selects evaluation words over cognition, whereas the other 
two vocabulary types are barely used. (3)Combination type includes two kinds of circumstances of vocabulary 
use. It covers those who use nearly equal amount of cognition and evaluation words, and the rest two vocabulary 
types are scarcely used. Secondly, reflective authors can be also labeled as combination type if three or more 
types of vocabulary are found, and each type should be responsible for over 10% of word use. 

Initially, the categorization of reflection was determined by the degree to which each vocabulary type (emotion, 
memory, cognition, or evaluation). It was discovered, however, emotion and memory words were nearly 
invisible in students’journals. In light of this, we then proposed a 3-category scheme consisting of cognition, 
evaluation and combination in which emotion and memory were not put in. Table 4 shows the distribution of the 
number of learners for reflection categories, where the cognition category accounts for the largest percentage, 
which coincides exactly with the result proposed by Lee (2002) and Lin (2004). 
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Table 4: Distribution of the number of learners for reflection categories 


Reflection category 

Number of learners 

Percentage 

Cognition 

24 

53.3 

Memory 

0 

0.0 

Emotion 

0 

0.0 

Evaluation 

9 

20.0 

Combination 

12 

26.7 


Effects of Reflection Categories on Learning 

The Levene’s tests of variance homogeneity on the three categories of reflection for the three learning outcomes 
(achievement test, work and attitude) are all not significant, meaning the variances between the three categories 
of reflection for the three learning outcomes are all the same. The result coincides with the assumption of 
MANOVA. Table 5 shows that the achievement test and work scores of the cognition category of learner 
reflection are all greater than those of the other two categories of learner reflection. Likewise, the attitude score 
of the evaluation category of learner reflection is greater than that of the other two categories of learner 
reflection. However, Table 5 shows that their significant levels are all not reached, meaning that reflection 
category does not influence achievement test, work and attitude. 


Table 5: MANOVA of learning outcome for different categories of reflection 


Wilk’s 

Learning 


Mean (SD) 


F 

Sig. 

Effect size 

A 

outcome 

Cognition 

Evaluation 

Combination 

0.863 

Achievement 

test 

59.38(18.96) 

53.00(14.83) 

56.11(19.00) 

0.256 

0.776 

0.019 

(0.702) 

Work 

74.88( 6.01) 

73.60( 3.36) 

72.56( 5.90) 

0.496 

0.614 

0.035 

Attitude 

63.12(7.94) 

64.00( 5.24) 

57.67 (9.80) 

1.519 

0.237 

0.101 


Effects of Reflection qualities on Learning 

Based on the quartile approach (Ql=65.5, Q3=79), the learner reflection qualities were divided into high scores 
group (front 25%), middle score group, (middle 50%) and low score group (rear 25%). The Levene’s tests of 
variance homogeneity on the three groups of reflection quality for the three learning outcomes (achievement test, 
work and attitude) are all not significant, meaning the variances between the three groups of reflection quality 
for the three learning outcomes are all the same. The result coincides with the assumption of MANOVA. The 
result of MANOVA reveals (Table 6, Wilk’s A= 0.449 and p<0.01), there is a significant difference in at least 
one learning outcome among the three groups of reflection quality. Furthermore, the difference in achievement 
test (p=0.005), work (p=0.006), and attitude (p=0.025) among the three groups of reflection quality are all 
significant, meaning reflection quality influences achievement test, work and attitude. According to post multiple 
comparisons among different groups using Scheffe’s approach (Table 7), the achievement test, work and attitude 
of the high score group of reflection quality are all greater than those of the low score group of reflection quality. 
This result reveals that reflection quality has positive effects on learning outcome. 

According to the effect sizes, the effects of reflection quality on the three kinds of learning outcome in a 
step-down sequence are achievement test, work, and attitude respectively. The three kinds of effects are all 
significant, while the effects on achievement and work are quite approximate. 


Table 6: MANOVA of learning outcome for different groups of learner reflection quality 


Wilk’s 

Learning 

outcome 


Mean (SD) 



Sig. 

Effect 

A 

High 

Middle 

Low score 

F 

size 



score 

score 





0.449 

Achievement test 

72.78(17.87) 

50.36(14.34) 

51.43(13.14) 

6.621 

0.005** 

0.329 

(0.002* 

Work 

78.56( 4.36) 

72.57( 3.34) 

70.86( 7.24) 

6.270 

0.006** 

0.317 

*) 

Attitude 

66.22(10.05) 

61.93(6.01) 

55.14 (6.67) 

4.248 

0.025* 

0.239 


*p<0.05, **p<0.01 


Table 7: Post multiple comparisons of learning outcome among different groups of learner reflection quality 
Learning outcome Post multiple comparisons (Sig.) 

Achievement test High score group > Middle score group (0.007**) 
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Work 

_ Attitude 

*p<0.05, **p<0.01 


High score group > Low score group (0.033*) 
High score group > Middle score group (0.023*) 
High score group >Low score group (0.013*) 
High score group > Low score group (0.025*) 


DISCUSSION 

Regarding the effect on learning outcomes, the reflection category has no significant effect on the three types of 
learning outcome, but the reflection quality has a significant effect on the three learning outcomes. On effect 
sizes, the effects of the reflection quality on the learning outcome, in descending order, are achievement test, 
work, and attitude. 


The reflection categories concluded in this study are cognition, evaluation and combination. In case of more 
categories, the reflection category may possibly have effects on the learning outcome. According to the study by 
Lee (2002), there are significant differences between learners of different reflection categories in understanding 
of chemical concept. Whether it can be attributable to different reflection categories concluded by Lee requires 
further studies. As a learner writes in his portfolio: 

"I never wrote reflection like this before. I have ever written diary before, but it seems that the reflection written 
in a portfolio is far different from diaiy. The diary I wrote before often lack meaningful reflection. Although lam 
not sure what kind of reflective learner I am, I can make sure that the reflection written in my e-portfolio is 
beneficial to my learning. No matter what type of reflection I wrote, I think it (reflection type) is not associated 
with my learning outcomes. ” 

Reflection involves a learner’s introspection and comments. Deeper reflection is helpful for learning (Blatt, 
Plack, Maring, Mintz, & Slmmens, 2007; Etkina, Karelina, Ruibal-Villasenor, Rosengrant, Jordan, & 
Hmelo-Silver, 2010; Gibson, Bernhard, Kropf, Ramirez, & Van Strat, 2001; Masui & De Corte, 2005). Therefore, 
it makes sense that reflection content has significantly positive effects on learning outcome, which is consistent 
with results from other studies that might focus on different fields of discipline or subjects. In attitude, the use of 
portfolio assessment may strengthen the students’ motives for learning (McAlpine, 2000). In learning outcome, 
writing reflection journal can enhance scientific literacy of pre-service teachers (Gibson, Bernhard, Kropf, 
Ramirez, & Van Strat, 2001). Reflection will also boost self-learning effectiveness (Yancey, 2001). As a learner 
writes in his portfolio: 

“While writing reflection, I think of whether there is a room for improvement on my learning situation, learning 
attitude, works, and test scores. After finishing the writing, occasionally I will tty to improve myself based on the 
reflection, hoping to make my learning outcome better than ever Therefore, reflection is somewhat helpful for my 
learning 

CONCLUSION AND IMPLICATION 

Conclusions derived from this study are illustrated as follows. During the course of implementing the Web-based 
portfolio assessment, the effect size of reflection quality on learning outcomes was small but significantly 
positive. Follow-up contrasts found reflection quality significantly related to achievement test, work, and attitude 
outcomes. These results may serve as a reference for future researchers and teachers who are engaged in studies 
or teaching related to Web-based portfolio assessment. Through Web-based portfolio assessment, this study can 
also help learners boost their learning reflection skills so that they may become self-reflection practitioners in the 
future. 


As the mental development of junior high school students is not yet completed and they lacked experience in 
writing reflection in previous courses, the learners’ reflection content is either insufficient or superficial or they 
use ambiguous words, although an outline of the writing has been prompted and demonstrated in advance. Since 
most learners had no experience of self-assessment and peer-assessment before, the teacher has explained the 
scoring method and precautions, despite that, there is still possible unfair assessment or inadequate assessment 
capability. Moreover, because there are quite a few questions in the portfolio assessment questionnaire, each 
student has to rate a number of his or her peers, resulting in a heavy burden to the students. These are limitations 
of this study, which need to be eliminated as far as possible in future studies. 

It is found in this study that reflection quality will affect achievement test, work and attitude; therefore it is 
necessary to reinforce the students’ reflection writing capabilities. According to the recommendation proposed 
by Falls (2001), students may write reflection based upon assessment rubrics of reflection. In addition to 
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convenience for writing reflection, it allows reviewing students’ learning process and learning behavior easily 
from their reflections. Also, teachers may encourage students to review peer reflection as possible so they can 
observe and learn peers’ reflection writing skills to improve their writing capabilities. Moreover, by reviewing 
peer reflection, the students may understand peers’ learning processes and outcomes and serve as a stimulus for 
learning. 

Falls (2001) argues that when students are writing reflection, the teacher should teach them how to write 
reflection and discuss writing skills, and students need more time to practice writing sophisticated reflection. 
From students’ portfolios, we find the content of their reflections is not profound enough, therefore the reflection 
activities require more refined demonstration, guidance and support from teachers. As Stone (1998) suggests, it 
is difficult for students to write reflection, because reflection is a process needing assistance and guidance. 

Despite the Web-based portfolio assessment system provides outlines for writing reflection and the teacher have 
explained reflection writing strategies, which have dramatically helped students write reflection, however, there 
are still a few students who use insufficient words when writing reflection or use inappropriate words in their 
reflection. If the system offers suitable or frequently used words or phrases, not only the writing of reflection 
will be improved but also the quality of reflection will be further boosted. 

This study induces categories of reflection simply based on the words and phrases in the reflection, without 
in-depth exploration of reflection through the content analysis or discourse analysis. What messages are shown 
in reflection? Do the messages reflect the students’ introspection? Can they reflect the students’ improvement in 
learning? These are all critical issues that should be addressed in follow-up studies. 
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